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Spike patterns in vivo are often incomplete or corrupted with noise that makes inputs 
to neuronal networks appear to vary although they may, in fact, be samples of a 
single underlying pattern or repeated presentation. Here we present a recurrent spiking 
neural network (SNN) model that learns noisy pattern sequences through the use of 
homeostasis and spike-timing dependent plasticity (STDP). We find that the changes in the 
synaptic weight vector during learning of patterns of random ensembles are approximately 
orthogonal in a reduced dimension space when the patterns are constructed to minimize 
overlap in representations. Using this model, representations of sparse patterns maybe 
associated through co-activated firing and integrated into ensemble representations. 
While the model is tolerant to noise, prospective activity, and pattern completion differ 
in their ability to adapt in the presence of noise. One version of the model is able to 
demonstrate the recently discovered phenomena of preplay and replay reminiscent of 
hippocampal-like behaviors. 



Keywords: spiking, STDP, learning, sequences, prospection, preplay, replay, memory 



INTRODUCTION 

The CA3 region of the mammalian hippocampus is a typical 
example of a recurrent neural network in vivo (Hasselmo et al, 
1995; Rolls, 2000; Kobayashi and Poo, 2004; Kesner, 2007; Li 
et al., 2010). It is known these recurrent networks are well-suited 
for learning pattern sequences and pattern completion (Treves 
and Rolls, 1992; Hopfield, 1995; Kali and Dayan, 2000; Gold and 
Kesner, 2005; Leutgeb et al, 2007; Yassa and Stark, 2011). While 
in general neural networks (Graves et al., 2009) have shown their 
ability to perform tasks on highly corrupted data, spiking neural 
network (SNN) models are often sensitive to input scale and must 
be carefully tuned to generate the desired output (Buonomano, 
2005). These issues are amplified in recurrent SNNs where insta- 
bilities can result in cascades of activity even with slight input 
perturbations (Gerstner and Kistler, 2002). Furthermore, biologi- 
cal networks are exposed to a great degree of input variability that 
can cause many simulated SNNs to fail. 

In order to address input variability, recent model implemen- 
tations showed that it is possible to globally scale the synaptic 
weight update on afferent synapses so as to constrain the cumu- 
lative weight to an artificial limit. This helps to also maintain 
stability in the model (Song et al, 2000; Van Rossum et al, 
2000). However, presynaptic spike timing is primarily main- 
tained locally at each synapse without evidence in vivo for an 
instantaneous global rescaling of all afferent synaptic weights. 
One biologically plausible solution is to incorporate homeo- 
static regulation. Several forms of homeostasis exist in biologi- 
cal networks that occurs at many timescales and is critical for 
the stability of these networks (Malinow and Malenka, 2002; 
Renart et al, 2003; Turrigiano and Nelson, 2004; Deeg, 2009; 
Turrigiano, 2011). Moreover, modeling homeostasis allows for a 



self- adjustment and overall scaling of input synaptic weights to 
neurons in a network and can help compensate for this input 
variability. 

Learning of pattern sequences through the use of plasticity 
has been studied before (Berthouze, 2000; Arthur, 2006). These 
methods have relied on learning associations with sequences of 
on- going reliable and repetitive background activity. However, 
repeatable sequences have only been detected during task- related 
activity (Itskov et al, 2011). Moreover, the sequencing of activity 
(Diba and Buzsaki, 2007) is likely to be based on self-sustaining 
recurrent pathways of activity in the network due to previous 
association with sensory and spatial activity. 

In this paper, we present three variations of a recurrent SNN 
for learning spatiotemporal patterns of activity using spike- 
timing dependent plasticity (STDP) with homeostatic regula- 
tion of activity. This model can learn to integrate patterns into 
pattern sequences from multiple noisy presentations and com- 
plete previously learned patterns from partially available data. 
Furthermore, stable pattern learning can be achieved with synap- 
tic weight changes despite relative differences in intensity of the 
input patterns. The network demonstrates both prospective fir- 
ing of activity and pattern completion while maintaining stability 
without the need to balance the input weights directly. The lessons 
learned from the development of these models are discussed 
and potential opportunities for handling their limitations are 
noted. 

MATERIALS AND METHODS 
SIMPLE NETWORK MODEL 

The simple network model is comprised of a recurrent popula- 
tion of Izhikevich neurons (Izhikevich, 2007a) with the default 
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parameters from Table 1. Each neuron connects to a fraction 
(pe and pi) of all other excitatory and inhibitory neurons (see 
Figure 1) with both excitatory and inhibitory connections having 
a random delay (xe delay and ti delay ). The network is constructed 
such that for each excitatory connection, an inhibitory connec- 
tion is also made between those neurons. For simplicity, one 
neuron is used for both connections, however, in an alternate 
larger model with different input scheme these dual paths are 
implemented with separate inhibitory and excitatory neurons. 
The recurrent network structure itself is random but balanced 
such that each neuron has an equal number of efferent and 
afferent connections and all neurons have the same number of 
connections. 

Neuron dynamics (Izhikevich, 2007a) are described by three 
variables, membrane voltage (v), recovery (u), and excitatory 



Table 1 | Simple model parameters. 



Network attribute 


Parameter 


Value 


Synaptic conductances 


Ej 


—81 mV 




c 

t e 


n m\/ 
U 1 1 1 V 




x / 


50 ms 




x e 


35 ms 




delay 
x / 


[1, 5] ms 




delay 
x e 


[11, 16] ms 




w max 


0.8 nS 




w min 


0.5nS 




w max 


0.8 nS 




^max 


OnS 


Excitatory triplet STDP 


LTP 

T 


20 ms 




x e T ° 


25 ms 




5 LTP 


0.020 nS 




S LTD 


0.024 nS 


Inhibitory top hat STDP 




40 ms 




XL™ 


700 ms 




5 LTP 


0.02 nS 




S LTD 


0.05 nS 


Homeostatic excitatory scaling 


"Ceo 


600 s 






[0.25, 0.35] Hz 


1-to-1 simple network 


A/a, 


480 




Pi 


1 




Pe 


1 


Neuron model 


a 


0.02 




b 


0.5 




c 


-40 mV 




d 


55 




v, 


-45 mV 




V r 


-60 mV 




V P 


40 mV 




c 


50 pF 




k 


0.5 


Noise 


V 


0.05 Hz 



scaling of afferent synaptic conductances (s, long-term homeo- 
static parameter): 



V = ^( k ( V ~ y r)( y -V t )-U + I syn + /noise) (1) 



u' = a(b(v — v r ) — u) 



(2) 
(3) 



where I no ise is zero mean gaussian noise with a = 80 pA, a)£> is the 
target firing rate, and x w is the time constant. I syn is composed of 
inputs from N excitatory and M inhibitory synapses: 



N M 
/'=! k=l 



(4) 



where gj and g\ are the excitatory and inhibitory synaptic conduc- 
tances respectively Upon action potentials (v > vp), 



v = c, u = u + d, and s = s 



(5) 



Integration was performed using Euler's method for all variables 
[except v where a hybrid method (Izhikevich, 2010) was used] 
with a simulation time step of 0.5 ms. 

The network model is designed to have inhibitory and excita- 
tory connectivity between neurons so that the effective pairing 
between any group of co-active neurons (an ensemble) and 
another neuron scales from net inhibitory to net excitatory for a 
given ensemble. Therefore, a direct connection from a neuron to 
another neuron maybe strongly excitatory, while the indirect con- 
nectivity between the presynaptic and postsynaptic neuron maybe 
inhibitory depending on the currently active ensemble. A network 
with the aforementioned properties can then associate positively 
or, by default, negatively each neuron with each ensemble. As long 
as the coding of the inputs and relative connectivity of the net- 
work spread the representation of ensembles such that these net 
effects from ensembles to any neuron can be modulated, new pat- 
terns can be learned while minimally effecting the coding of other 
ensembles (see Results). Thus, these networks exhibit the ability 
to learn or recall spike patterns whether they code for sequential 
spatial, sensory, or other data. 

Two different forms of synaptic plasticity are used for the 
model, one for excitatory and one for inhibitory synapses. The 
inhibitory synapse rule is symmetrical, and functionally imple- 
ments a rule where co- active neurons reduce their inhibitory cou- 
pling, but neurons that fire independently have strong inhibitory 
connectivity. Inhibitory plasticity uses an inverted top -hat shaped 
symmetric STDP curve, which is similar to a Mexican-hat plas- 
ticity curve (Caporale and Dan, 2008; Srinivasa and Jiang, 2013). 
Upon presynaptic or postsynaptic action potential: 



Aw; 



8™ 

4 tp 

0 



if k 



post 

else if 1 1 



^pre 



< X 



LTD 



post 

otherwise 



^pre 



< X 



LTP 



(6) 
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FIGURE 1 | Recurrent and input network structure and distribution. 

Recurrent connections are shown only for the center recurrent neuron. 
(A) Input structure of the simple model with 1-to-1 network inputs, recurrent 
neurons receive input from one and only one input neuron. (B) Input 
structure of the alternate model with many-to-many network inputs, each 
recurrent neurons receives input from 12 random input neurons (which 
project 3 outputs each). (C) Unbalanced usage of the input space with and 
without input neuron allotment. Histogram shows the distribution of the 



number of input spikes due to a specific neuron. (C) Shows the activity of 
480 input neurons when the patterns are chosen consist of separate groups 
of neurons used for simple model inputs. (D) Show the multimodal nature of 
the 8000 input neurons' activity when neurons are selected randomly 
without regard for usage in other ensembles, as is the case in the alternate 
model. The leftmost mode represents noise, while the successively 
rightward modes of the distribution represent redundant usage of the input 
neurons in the episode patterns. 



where w is constrained to 0 < W{ < w™ ax , t™ is the time win- 
dow of long-term depression (LTD) for inhibitory STDP. The 
parameters o^ TD and o^ TP correspond to the change in synaptic 
weights for inhibition during LTD and long-term potentiation 
(LTP), respectively The terms f pre and £ pos t correspond to the time 
at which presynaptic and postsynaptic spike events occur. 

Complementary to the inhibitory weight changes, excitatory 
changes are asymmetric and strengthen synapses that contribute 
to the causal activation of the postsynaptic neuron and weaken 
those that are activated in reverse order unless the postsynaptic 
neuron activates again. Excitatory plasticity follows a triplet- 
based STDP rule (Pfister and Gerstner, 2006) and is described by 
the dynamics when presynaptic or postsynaptic action potentials 



Aw e 



(8™ + Aw™)e~ 



iff. 



post 



tpre > 0 



(7) 



8™6 



iff. 



post 



^pre 



< 0 



where w is constrained to 0 = W{ = w™ ax and Ai^ TD is the 
change due to the last depression event. The parameters 8^ TD and 
8g TP correspond to the change in synaptic weights for excitation 



during LTD and LTP, respectively and x|r TP is the time window of 
LTP for excitatory STDP. Excitatory synaptic weights are initial- 
ized with values from a uniform distribution on the interval [0, 
0.1 w™ ax ). 

ALTERNATE NETWORK MODEL 

An alternate model of 2000 excitatory and 500 inhibitory neurons 
was used for some simulations with several other modifications 
detailed below. Excitatory neurons were recurrently connected 
with other excitatory neurons with probability pee and with the 
inhibitory neurons with probability pei. Inhibitory neurons were 
recurrently connected with other inhibitory neurons with proba- 
bility pii and with the excitatory neurons with probability pie, see 
Table 2. 

The alternate model used Izhikevich's fast spiking (FS) model 
for inhibitory neurons and increased the membrane capaci- 
tance, C, of the neuron model to 500 pF for the excitatory 
neurons. Furthermore, the excitatory reversal potential, Ee was 
increased to 40 mV to prevent excitation lock that could occur 
with the increased membrane capacitance. Excitatory- excitatory 
and excitatory- inhibitory connections used the aforementioned 
triplet STDP rule, while inhibitory- excitatory connections used 
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Table 2 | Alternate model parameter modifications. 



Network attribute 


Parameter 


Value 


Network structure 


N e N 


2000 




N'n 


500 




Pa 


U.o 




Pie 


0.3 




Pei 


0.3 




Pee 


0.3 


Neuron model 


c 


500 pF 




Ee 


30 mV 



the inverted top-hat STDP rule and inhibitory- inhibitory connec- 
tions used a non-inverted top-hat STDP rule. These inhibitory 
connections were also changed to an order of magnitude faster 
conductance decay time constant. A single inhibitory feedback 
neuron with long decay time constant was added with connec- 
tions to and from all excitatory neurons; these connections were 
non-plastic. 

PREPLAY MODEL 

Patterns of place cell activity are preplayed in forward order or 
replayed in reverse order at the beginning and end, respectively, 
of a linear track (De Almeida et al., 2007). Similarly, one ver- 
sion of our model demonstrates preplay and replay of activity. 
Although less biologically plausible, this model used the same 
input and recurrent connection scheme and was the same size as 
the simple model with a few exceptions. Firstly, each neuron was 
the regularly spiking simple four parameter model (Izhikevich, 
2003) but with membrane capacitance lOOOx greater, the a 
parameter lOOOx smaller, and the d parameter 10 x smaller. 
Secondly, weight changes were larger with an inverted window for 
inhibitory STDP (this was allowable since the network was more 
stable due to higher capacitances and slower postspike recovery). 
The parameter settings for this model are summarized in Table 3. 

NETWORK INPUT 

Two network input structures were considered: a 1-to-l input 
neuron to recurrent neuron model and a many-to-many input 
model where input neurons spread output onto three recur- 
rent neurons and each recurrent neuron receive 12 connec- 
tions from input neurons. 1-to-l network inputs are used with 
the simple model and preplay model; the many-to-many net- 
work input structure is used with the alternate model. In the 
simulations with 1-to-l inputs, each neuron receives simu- 
lated spiking input through a single large, non-plastic, excita- 
tory synapse from an input neuron so that a spatiotemporal 
episode can be strongly forced onto the downstream recur- 
rent network model. This strong influence results in the recur- 
rent network having a high likelihood to spike in a similar 
manner as the upstream inputs. This input drive is modeled 
as noisy frequency modulation of input spikes as shown in 
Figure 2. 

Twelve episodes were constructed each consisting of 25% of the 
network and segmented into 30 temporal steps. These episodes 



Table 3 | Preplay model alterations. 



Network attribute 


Parameter symbol 


Value 


Time constant for recurrent inhibitory 




65 ms 


synaptic conductances 






Time rnnctant for ron irront Dvrita+nr\/ 
i line L/Uiioiciiii iui i cuu 1 1 ci 1 1 caui id iui y 




1 20 ms 


synaptic conductances 






Time window for LTP portion of the 




140 ms 


inhibitory STDP function 






Time window for LTD portion of the 




56 ms 


inhibitory STDP function 






Excitatory reversal potential of neuron 


E e 


30 mV 



were presented in blocks of four episodes each and consisting of 
100% of the network for the 1-to-l network input structure. In 
each block, the episodes were presented at random (from a uni- 
form distribution) and with overlap of four temporal steps. The 
current step is presented with the last few patterns in decreasing 
intensity each low frequency period. Neurons comprising each 
temporal step are activated simultaneously (with one high fre- 
quency period of jitter) for each active spatiotemporal step. As 
each episode is presented, one neuron out of each temporal step 
is selected at random to be removed from that presentation of the 
episode. 

In the alternate model, the episodes are encoded in the same 
manner. However, the input neuron allotment to each episode 
was performed without regard for overlap and redundant usage 
during each group of episodes (see Figure 1). Additionally, a 
larger network was used and as a result each temporal step 
consisted of 14 neurons as opposed to 4. 

In the preplay model, during each episode presentation each 
temporal step was repeated 7 times before proceeding to the next 
temporal step, whereas in the simple and alternate models each 
step occurred once per episode. 

For all models, noise was injected into the network by adding 
spontaneous action potentials from the input neurons at a rate 
of v = 0.05 Hz per neuron in addition to the somatic gaussian 
current input. A single, fast inhibitory, input neuron projects to 
the entire recurrent network with a fixed, but varied time delay 
(1-4 ms). The inhibitory input neuron supplies the network with 
high frequency modulated action potentials at a rate of 250 Hz. 

SIMULATION AND ANALYSIS 

Simulations were performed using a custom C/C++ MPI- 
based simulator and run 2-9 x real time on a 2 GHz 
quad-core i7 MacBook Pro with 8 GB of RAM. Spike data 
and weights were analyzed using MATLAB. Local lin- 
ear embedding (LLE) of weights was performed using the 
MATLAB Toolbox for Dimensionality Reduction freely avail- 
able at http://homepage.tudelft.nl/19j49/Matlab_Toolbox_for_ 
Dimensionality_Reduction.html 

RESULTS 

The results are summarized for the models as follows: the simple 
model with the complete usage and non- overlapping allotment of 
input neurons, the alternate model with the random allotment of 
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FIGURE 2 | Pattern of noisy but structured network inputs. Every 1/7 of a 
second (low frequency), part of a spike episode is input to the network. Each 
episode is split into 30 temporal steps, each associated with a few input 
neurons. Groups of four temporal steps are presented sequentially in time (in 
successive high frequency, 1/49 s, blocks) with the current step being 



presented at the middle of the low frequency cycle (where the darkness of 
the high frequency blocks drawn above relate to the increased probability of a 
spike). In each successive low frequency block, the current temporal step is 
advanced by one, similar to activity reported in Maurer and McNaughton 
(2007). 




0 12 3 4 

Time (sec) 



FIGURE 3 | Prospective spiking activity in the simple model. Example 
raster plot of activity in the recurrent network after training of an episode in 
the 1-to-1 network. Neuron have been resorted by the mean time of activity 
in the interval. Spikes near and to the left of the blue line are, in general, 
prospective activity. 



inputs, and the preplay model with varied input types. In most 
cases, inputs are partially presented, noisy, and probabilistic (in 
their occurrence) resulting in a disproportionate use of input neu- 
rons (Figures 1, 2). It was necessary to use the alternate input 
mapping when using random allotment to prevent the growth of 
homeostatic scaling and bursting (and recruitment of the entire 
network) in the recurrently connected neurons not assigned to an 
ensemble (data not shown). This connection strategy ensures that 
most recurrent network neurons will receive some input, since the 
fraction of neurons without active inputs is small. Additionally, a 
many-to-many input architecture amplifies the amount of noise 
by increasing the number of input noise sources to each neuron in 
the recurrent network, which prevent the uncontrolled growth of 
the homeostatic parameter through direct activation of the neu- 
ron and indirectly by promoting random changes in its weights 
that may result in the "unused" neuron becoming active in a 
random ensemble without direct stimulation. 

LEARNING EPISODES 

The use of low and high frequency modulated inputs force 
short, high frequency bursts of spikes that replicate a sequen- 
tial long time scale procession in temporal steps. After learning, 
the sequential inputs activate subsequent nodes prospectively 
(see Figure 3). The net inhibitory effect of on-going firing lim- 
its prospective firing to a handful of future temporal steps. The 
network demonstrates a cascade of activity during each low fre- 
quency cycle. This cascade is preempted by the intermittent high 
frequency inhibition and terminated by an increase in low fre- 
quency inhibition and due to refractoriness of the neurons. The 
beginning of the next low frequency cycle is marked by the input 
stimulus that excites another cascade of activity. 

Activity in the network is initially the direct result of activated 
input neurons whether from noise or as part of the episode. As 
the network adapts to the input episodes, the background noise is 
suppressed and missing components of the repeated episode are 



activated along with the neurons that are soon to be activated. 
Background noise is suppressed due to recurrent inhibition and 
the down regulation of input scaling so as to counter the up reg- 
ulation of recurrent connections between members of ensembles. 
The co -activation of neurons over several presentations allow 
for the learning of an episode on the basis of partial activity 
of member neurons in the ensemble when the partial pattern is 
reactivated as shown in Figure 4. 

With 1-to-l input structure, the simple network neurons are 
excitable relative to their history of activity as expected. With 
the many-to-many input structure, the alternate network neurons 
that are active in fewer ensembles are excited stronger than other 
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neurons in an ensemble. Neurons become active prospectively 
firing prior to the presentation of input that occurs in subse- 
quent temporal steps. Neurons, also, complete missing portions 
of the patterns presented. The amplitude of the pattern comple- 
tion and prospective activity is less than that of neurons directly 
activated by episodic input (see Figure 5). The prospective and 
pattern completion firing rates formed a smooth curve while the 
directly stimulated activity was far greater. This leads to the notion 
that the prospective firing and pattern completion are related by 
a common mechanism in this model, and that the prospective 
activity is pattern completion of the representation forward in 
time. However, see Noise Effects for differences in the learning 
of these phenomena. 

SYNAPTIC WEIGHT CHANGES 

To examine learning at the synapses, LLEs of the excitatory and 
inhibitory weight spaces were performed over the course of learn- 
ing. Briefly, LLEs map high dimensional spaces into neighbor- 
hood preserving low dimensional spaces (Roweis and Saul, 2000). 
The globally mapped nature of the low dimensional embedding 



makes them appropriate for visualizing data and relative tra- 
jectories in high dimensional spaces. Excitatory and inhibitory 
dimensionality reductions were performed separately due to the 
different time constants and learning rules involved. L2 distance 
in the weight space served as a poor indicator of learning (data 
not shown) and asymptotically demonstrated strong relation to 
the simulation time difference between weight vectors during 
learning. 

Instead, the distance in a few dimensions of LLE space 
appeared to be a better indicator of changes in the network. For 
example, in a representative example, comparing three sequen- 
tial sets of learning in the LLE space results in the expected 
non-linear adaptation and convergence of weights of recurrent 
network weights in the simple model. In this 3D LLE, these adap- 
tations move along three nearly linear trajectories, each of which 
are nearly orthogonal (see Figure 6). This shows that given a 
global embedding into a low dimensional space, the weight vec- 
tor trajectory moves from neighborhood to neighborhood in such 
a way that global changes are not discordant. During adapta- 
tion, the targeted correlated activity reduces inhibition, which in 



Recurrent Neurons 



Recurrent Neurons 



Recurrent Neurons 




FIGURE 4 1 Activity in alternative network model. Instantaneous 
firing rate of the input neurons and recurrent network during an 
episode with m any-to-many connections. Smoothed firing rate (binned 
in 1/7 s and smoothed with a causal exponential kernel, x=1/7s, 
over 3 bins) demonstrates an example of recurrent network activity 
(right) leading the input neuron activity (left) throughout the 



progression of an episode near the (A) beginning, (B) middle, and 
(C) end. Recurrent neurons are matched to each input neuron and 
are redundantly represented for each input neuron they receive 
synapses from leading to the appearance of more background noise. 
Color scale ranges from dark blue to dark red representing a 
0-20 Hz firing rate, respectively. 
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FIGURE 5 I Pattern prospection and completion in the simple model. 

Instantaneous spike firing rates (in 1/7 s window) relative to the current step 
in the episode for the (A) simple model, (B) preplay model, and (C) preplay 
model with slower progressing stimulation with increased overlap. Negative 
lags are yet to be stimulated. Green triangles, black squares, and blue circles 



represent the average firing rate of recurrent neurons that are soon to be, 
currently being, and are not but would normally be stimulated via input 
neurons. Thus, they highlight prospective, active, and pattern completing 
neural activity, respectively. Bars indicate the 5-95% percentiles of firing 
rates for each marker. 
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FIGURE 6 | Dimensionality reduction of the model weight space. 

Reduction of the simple model 230,400 weight space [of (A) excitatory 
and (B) inhibitory recurrent synapses] to 3 dimensional space using local 
linear embedding. Black points are the initial state of the weights. Red 
points indicate the learning of the first group of 4 patterns. While green 
and blue represent the learning of the second and third patterns, 



respectively. Note that the weight changes are nearly orthogonal in the 
reduced dimensional space. There is some effect in the neighborhood 
finding process that blends between neighboring points (and thus 
episodes) at the tail of each segment of the episodic learning except the 
last one. (C,D) Excitatory and inhibitory, respectively, weight space 
reduction for the preplay model. 



turn enables faster spiking and thus faster weight changes (see 
Figure 7). Weight adaptation rate peaks and slows as the pattern 
is learned and weights reach their maximum or minimum values. 
This coincides with a peak in the combined degree of prospectiv- 
ity (i.e., look ahead time window) and pattern completion ability 
of the network (see Figure 5). Increasing the number of episodes 
(redundant allocation) that were being learned resulted in the tra- 
jectory of the learning in the first three LLE dimensions to appear 
less orthogonal (data not shown). This implies that learning of 
the groups of patterns does not interfere with each other when 
the subsequent uses of neurons are in completely new ensembles; 
however, pairwise reuse of neurons within the STDP windows 
violates this condition. 

NOISE EFFECTS 

To define a metric to quantify network prospectivity and pat- 
tern completion, the firing rate of neurons for up to 2 tem- 
poral steps into the future of the episode or the neurons 
from the current step that were removed from the input are 



averaged and compared with the activity of the directly stimulated 
neurons. 

^(prospection | completion) — ^background f , 

M-' active — M-' background 

where |x is the mean firing rate. Given that the inputs are not 
active for prospection or completion, this relative activity mea- 
sure is expected to be less than 1 as homeostatic input scaling 
regulates neuron activity based on inputs as well as recurrent con- 
nections. Remarkably, the network is tolerant to large amounts of 
background noise for pattern completion but less so for prospec- 
tive firing (see Figure 8). In contrast, the noise has little effect on 
the initial learning rate, but affects the final quality of learning 
(see Figure 8). 

PREPLAY AND REVERSE REPLAY 

The preplay network began to demonstrate prospection sooner 
than other models (on the second presentation of an episode — 
however each step is presented 7 x longer than the other models 
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FIGURE 7 | Approximate learning curve of prospection in the simple 
model. Each group of episodes is shown aligned to the first presentation of 
that group at time 0. Learning for Group 1 was slower due to the initial 
network adjustment in homeostatic values and the settling of synaptic 
weights at limit values. Learning occurred at the same rate for the two 
subsequent groups. Each learning curve represents the mean performance 
across four different simulation runs with four noise levels (n = 4) within a 
group. The error bars represent the standard deviation in ^ pr0 spection for that 
group at various episode durations. 



presented), and, in general, produced better and more stable pat- 
tern completion and prospective activity than the other models 
presented in this paper. Although more divergent from biological 
underpinnings, this model was able to demonstrate preplay and 
replay of activity in the form of self-sustaining sequences in for- 
ward or reverse, respectively. This is accomplished by reactivating 
the ensemble at the beginning (preplay) or end (replay) of the 
spatiotemporal sequence after learning has occurred. The episode 
is reactivated by short gamma frequency bursts of 1-4 spikes from 
each neuron in the initial or end segment of the episode (two 
temporal steps were used). This could be considered reminiscent 
of the upstream spiking activity due to the sensory input at the 
beginning or end of a linear track. The preplay of activity (see 
Figure 9) can be viewed as a full recall of a previous memory. The 
recall of previous memories based on partial patterns means that 
the results of various behaviors can be predicted given a similar 
sensory match to other experiences. The model also exhibits var- 
ious successful durations of the preplay and replay of the entire 
episode that is qualitatively similar to the results to other recent 
research (Foster and Wilson, 2006; Diba and Buzsaki, 2007). 

However, there are several limitations of this version of the 
model. This model aggressively suffered interference and learned 
more slowly the second and third sets of patterns. The sequence 
preplay speed did not occur on biologic timescales. The activ- 
ity propagated at a much slower rate due to the high mem- 
brane capacitance resulting in long time constants and the slow 
inhibitory currents. The preplay propagation speed increased 
when using the fast stimulation protocol used for the other mod- 
els due to depression of inhibitory weights. The capacitance and 
the localized slow inhibitory currents proved to be the reason for 
stability in this model; reducing the capacitance, as in the other 



0.25 




0 2000 4000 6000 8000 

Simulation Time (sec) 



FIGURE 8 | Effect of noise on the learning. Performance of the simple 
model through learning with incomplete patterns is shown here. Solid lines 
are the metric, R, for prospection. Dashed lines are the metric, R, for 
pattern completion. Note, the similar slope of the curves initially as 
compared to divergence in the final value relative to noise, v. v is expressed 
in terms of per neuron noise spikes in Hz. A significant amount of noise 
tolerance exists considering the input signals are an average of 6 spikes per 
presentation of each episode (which equates to a brief instantaneous firing 
rate of 10 Hz but averages out to 0.3 Hz over the course of the learning 
trial). The error bars represent the standard deviation in either ft pr0 psection or 
^completion for various episode durations. The results also show that the 
network is more robust to noise for pattern completion (solid lines) 
compared to prospective recall (dashed lines). 
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FIGURE 9 | Preplay of an episodic spike pattern. Left, activation of the 
episode that was previously learned. Activity to the left of the blue line is 
prospective. Right, fast preplay of the episode to the right. 



models, results in less stability, yet, faster response times in the 
cascade or sequencing of activity. When forming symmetric exci- 
tatory connections between neurons (in the triplet STDP case, 
and needed for reverse replay), faster response times gives the net- 
work the ability to recruit a major portion of the network and 
this in turn leads to bursting activity in the network. A better 
method of achieving a balance between the stability- sustainability 
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FIGURE 10 | Relative effect of synaptic connections with two time 
constants. The temporal length of excitation can be controlled by changes 
in the weights of dual synapse with different time constants. Blue lines 
conductance trace examples at two weights for slow inhibitory time 
constant. Green line is an excitatory conductance trace. Vertical, dashed 
red and black lines show the time point at which inhibition begins to exceed 
excitation. 



trade-off in the model could come from compartmental models 
and more robust and targeted inhibition. 

The preplay model was only able to demonstrate replay when 
the triplet STDP rule was used in conjunction with a temporal 
blurring of the input patterns as opposed to the sharp proba- 
bility peaks seen in Figure 2. A preplay model with traditional 
couplet STDP could learn, form prospective activity, and demon- 
strate preplay, however, reverse replay could never be achieved 
even when using shifted or noisy windows (Babadi and Abbott, 
2010). Another interesting aspect of this model was the fleeting 
fragments of episodes were reignited (primarily forward and on 
the same time scale as preplay) when not driven by an episode 
pattern but still receiving random noise inputs (supplementary 
video available online). 

DISCUSSION 

We demonstrated recurrent SNNs capable of learning episodes 
operating with missing, noisy, and unbalanced data was demon- 
strated. This learning is demonstrated by the prospective firing of 
neurons attributed to subsequent stimuli and the completion of 
missing portions of the patterns. This prospective firing is excited 
by recurrent connections from neurons in the currently active 
ensemble. The active ensemble contains neurons that are com- 
ponents of temporally adjacent (both previous and subsequent) 
ensembles; however, the previous neurons were just active and are 
generally in a refractory state and moderately inhibited. 

These recurrent spiking models provide a means to store pat- 
terns and recall or even predict them given a previously encoded 
pattern (Lisman and Redish, 2009). In the many- to-many input 
coding of the alternate model, the input to recurrent network 
mapping results in a randomized and more distributed encod- 
ing. This spreads activation through the recurrent neurons to 
preferentially recruit neurons that are not utilized or rarely uti- 
lized. 

Our model maintains stability through strong high frequency 
inhibition to limit the network activity to be cyclic and by intro- 
ducing the input pattern such that its driving force terminates at 
the peak of the low frequency cycle enabling recurrent activity to 
trail-off in the second half of the cycle. Therefore, the network is 
stable as long as network activity is always decaying which can be 
ensured by limiting the upside of excitatory weights, the downside 
of inhibitory weights, and using a time constant for inhibition 
that is slower than excitation. This balance enables the scaling of 
weights to modulate, on a neuron-to-neuron basis, the duration 
of time for which the net recurrent activity is excitatory before 
turning inhibitory. Setting the weight ranges so that the maxi- 
mum duration is half of the low frequency cycle designates the 
spiking activity due to inputs to become inhibitory to the network 
and thus a stabilizing force (see Figure 10). 

Other models exist which relate learning and sequences of 
spiking activity (O'Keefe and Recce, 1993; Rao and Sejnowski, 
2001; Buonomano, 2005; Lisman et al., 2005). Another method 
proposed by Buonomano (2005), uses the scaling of the presynap- 
tic component of the weight to learn a time delay and sequence 
a spiking pattern. However, those authors admit the difficulty 
with learning multiple different patterns. This work does not use 
the pattern correcting and forward lookup circuitry proposed 



in Lisman et al. (2005). However, this simple recurrent network 
is able to do both. As a result though, there is a tendency in 
this model to bring temporal associations forward and back- 
ward in the sequence learning — a separate pattern correction and 
completion network may solve this issue. Regardless, a spiking 
network that can perform free recall of multiple episodes has yet 
to be demonstrated in a simulated network that first learned the 
multiple episodes concurrently. 

LINK TO BIOLOGY 

Recurrent neural networks have been found throughout the 
brain (Rao and Sejnowski, 2001; Kobayashi and Poo, 2004; 
Buzsaki, 2006). Specifically, our network employs neural archi- 
tecture design that can produce hippocampal-like behaviors 
including reactivation, preplay and replay (Pavlides and Winson, 
1989; Wilson and McNaughton, 1994; Louie and Wilson, 2001; 
Andersen et al, 2006; Rasch and Born, 2007; O'Neill et al, 2008; 
Dragoi and Tonegawa, 2010; Gupta et al, 2010; Buhry et al, 

2011) . This enables the network to automatically load balance 
densely coded downstream networks through homeostatic reg- 
ulation and sparse upstream coding. Furthermore, the low and 
high frequencies used here are similar to the theta and gamma 
rhythms found in the hippocampus (Buzsaki et al., 1992; O'Keefe 
and Recce, 1993; Skaggs et al, 1996; Penttonen et al, 1998; Bragin 
et al, 1999; De Almeida et al, 2007; Lenck-Santini and Holmes, 
2008; Pastalkova et al., 2008; Zhang et al, 2011; Penley et al., 

2012) . 

However, this network is appreciably simpler than the one 
found in the hippocampus, including the neural dynamics, learn- 
ing rules, and homeostasis. The 80:20 ratio for excitatory to 
inhibitory connections is lower than that found in the CA3 region 
of the hippocampus but similar to cortical areas (Buzsaki, 2006). 
Moreover, we believe that the memory capacity of this network 
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to be markedly smaller than of a more complex network where 
feedback inhibition is present in the form of multiple indepen- 
dent neurons and target specific regions of the principal cell's 
dendrites, soma, and axon. Also, the mammalian hippocampus is 
thought to encode through the sparse representation of the den- 
date gyrus (DG) and recall using a direct entorhinal cortex to CA3 
pathway that contain several smaller connections as opposed to 
the DG-CA3 pathway (Treves and Rolls, 1992; Nolan et al, 2011) 
which was adopted here. 

FUTURE WORK 

Stability in this model is fragile with large weight changes, mean- 
ing that increased excitation or increased recruitment can easily 
lead to cascades of activity that result in bursts like many recur- 
rent networks (Wagenaar et al., 2006). Once the network bursts, 
the network continues to burst due to both the positive feed- 
back effect of triplet STDP and the negative feedback effect of 
inhibitory STDP during bursting. Separating late and early phase 
LTP (Adams and Dudek, 2005) may prevent infrequent occur- 
rence of bursts from having these catastrophic ramifications. 
However, this may not be the case since an analogous develop- 
ment occurs in vivo; for example, tetanic hippocampal stimuli 
evolve into epilepsy (Sanchez et al., 2006). Alternatively, the use 
of reward-based learning methods may increase the speed and the 



specificity with which learning occurs by identifying new patterns 
and promoting learning via neuromodulation (Izhikevich, 2007b; 
O'Brien and Srinivasa, 2013). 

In the preplay iteration of this model, there is evidence that this 
rudimentary kind of network supports replay and preplay at com- 
pressed timescales (Diba and Buzsaki, 2007). Due to the use of the 
triplet based rule, symmetric excitatory connections are formed 
between neurons allowing for both the forward or reverse propa- 
gation of activity. However, the triplet rule does not promote the 
robust symmetric connections found in the hippocampus with 
short bursts of post-then-pre-synaptic spikes. We believe that the 
use of a more phenomenological rule (Markram et al., 2012) 
would promote further symmetry in learning and robust replay 
on par with preplay without requiring as much blurring and over- 
lap of spike trains. Further investigation of preplay is necessary to 
determine if the compression factor seen in vivo (Euston et al., 
2007) occurs as a natural result of propagating neural activity 
without theta frequency resets. 
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